TITLE OF THE INVENTION 



DATA EDITING APPARATUS AND METHOD 



5 FIELD OF THE INVENTION 

The present invention relates to data editing 
apparatus and method, and more particularly, to data 
editing apparatus and method preferably used for 
generating a digital album utilizing binary data such as 
10 still image data, audio data and moving image data. 



BACKGROUND OF THE INVENTION 
As digital still cameras and digital video cameras 
are widely used, the amount of so-called binary data 
15 such as still image data, moving image data and audio 
data, handled for domestic use is dramatically 
increasing. To effectively utilize these binary data, 
some search and management methods are required. 

For example, one method is using a digital album 
20 to arrange digital image data and browse the data. A 
digital album is an image display which looks like a 
silver-chloride photo album, having a digital image 
pasted on a background image called a template. 

In use of digital album, image data can be simply 
25 arranged, and storage and browsing can be made without 
consideration of storage space of image data, and 
further, appearance of browsed images are improved. 

-1- 



However, actually, digital albums are not positively 
utilized for storage and management of digital image 
data. To generate a digital album, as it is necessary to 
design the album by e.g. selecting backgrounds of 
5 respective digital image data, the works to generate a 
digital album are complicated and troublesome. As the 
number of digital images increases, the load of the 
works increases. 

For example, moving image data obtained by image 

10 sensing using a digital video camera or the like can be 
inputted into a personal computer or specialized device 
and subjected to various processings and editing. 
Various processings and editing include addition of 
title, joining moving image data and the like. The 

15 moving image data are arranged by the editing, and 
appearance of moving image data upon browsing is 
improved. 

However, actually, people are not willing to 
improve an opportunity to edit moving image data since 
20 processing/editing of moving image data requires 

complicated works by using specialized software, and 
such work becomes heavy load on users. For example, it 
is troublesome to find necessary moving image data and 
it requires a skilled technique to rearrange data and 
25 process a joint between images. 

On the other hand, Japanese Published Examined 
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Patent Application No. Hei 11-234560 proposes a 
technique of adding meta data to still image data and 
audio data and utilizing the data for generation of 
moving image data. However, there is no technique of 
5 adding meta data to moving image data itself and 

utilizing the data for editing the moving image data. 

SUMMARY OF THE INVENTION 

10 The present invention has been made in 

consideration of the above situation, and has its object 
to appropriately assign binary data to previously- 
provided templates. This facilitates e.g. generation of 
digital album. 

15 Further, another object of the present invention 

is to automate moving image editing thereby facilitate 
editing of moving image data. 

According to the present invention, the foregoing 
object is attained by providingl a data editing method 

20 for performing editing on binary data by using plural 
templates having keywords, comprising: an assignment 
step of assigning each of plural binary data to one of 
the plural templates based on the keywords; and a 
construction step of, upon reproduction of the plural 

25 binary data by using the plural templates, constructing 
reproduction data so as to reproduce the plural binary 



data in accordance with the result of assignment at the 
assignment step. 

According to another aspect of the present 
invention, the foregoing object is attained by providing 
5 a data editing apparatus for performing editing on 
binary data, comprising: storage means for storing 
plural templates having keywords; assignment means for 
assigning each of plural binary data to one of the 
plural templates based on the keywords; and construction 

10 means for, upon reproduction of the plural binary data 

by using the plural templates, constructing reproduction 
data so as to reproduce the plural binary data in 
accordance with the result of assignment by the 
assignment means. 

15 Other features and advantages of the present 

invention will be apparent from the following 
description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 
the same name or similar parts throughout the figures 

20 thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 



The accompanying drawings, which are incorporated 
25 in and constitute a part of the specification, 

illustrate embodiments of the invention and, together 



with the description, serve to explain the principles of 
the invention. 

Fig. 1 is a block diagram showing the system 
configuration of a recording apparatus used for 
5 generation of digital album according to a first 
embodiment of the present invention; 

Fig. 2A is an example of data structure where meta 
data indicating the contents of image data is added to 
the image data; 

10 Fig. 2B is a schematic diagram showing image data 

and meta data indicating the contents of the image data 

linked to the image data; 

Figs. 3A to 3E are examples of meta data showing 

contents of image data 1 to 5; 
15 Fig. 4A is a table showing a digital album 

scenario; 

Fig. 4B is an example of the digital album 
scenario; 

Figs. 5A to 5D are examples of templates for 
20 generation of digital album; 

Fig. 6 is a table showing keywords linked to the 
respective templates; 

Fig. 7 is a flowchart showing a digital album 
generation procedure; 
25 Fig. 8 is a flowchart showing selection of 

template matching each image data; 
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Fig. 9 is a flowchart showing a procedure of 
inserting selected image data into a corresponding 
template; 

Fig. 10 is a block diagram showing the system 
5 configuration of moving image data editing apparatus 
according to a second embodiment of the present 
invention; 

Fig. 11 is an explanatory view of moving image 
editing processing according to the second embodiment; 
10 Figs. 12A and 12B are tables showing data 

structures where meta data is added to moving image 
data; 

Fig. 13 is a flowchart showing processing upon 
moving image editing; 
15 Fig. 14 is a flowchart showing moving image data 

search processing according to the second embodiment; 

Fig. 15 is a flowchart showing processing of 
display-describing moving image data in a template; and 
Fig. 16 is a flowchart showing the moving image 
20 data search processing by utilizing similar image search 
according to a third embodiment of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

25 Preferred embodiments of the present invention 

will now be described in detail in accordance with the 
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accompanying drawings . 

In the following embodiments, generation of 
digital album by using a recording apparatus will be 
described, however, the scope of the present invention 
is not limited to the embodiments. 

To perform image data search and management, it is 
efficient to handle image data including so-called meta 
data, i.e., additional information explaining the image 
data such as name of person, title of event, date and 
keyword. Meta data means "data on data" used as 
additional data explaining the contents of binary data 
such as image data and audio data. If the meta data is 
added to each binary data, image data, audio data and 
the like can be found by a keyword search. Further, as 
more meta data are added to the respective binary data, 
searches can be performed on the respective binary data 
with more keywords. In the following embodiments, meta 
data as described above is utilized. 

<First Embodiment> 

[System Configuration of Recording Apparatus] 

Fig. 1 is a block diagram showing the system 

configuration of a recording apparatus 100 according to 

a first embodiment of the present invention. 

In Fig. 1, a data input/output unit 106 

inputs/outputs various data including binary data such 



as image data with meta data, and meta data. An input 
unit 101 inputs instructions and data from a user by 
using various input devices such as a keyboard and a 
mouse . 

5 A display unit 102 which is a CRT, a liquid 

crystal display or the like, displays image data or 
displays a GUI screen image where picture, icons and the 
like are displayed. A storage unit 107 which is a hard 
disk is used for storing various data such as binary 

10 data and meta data. 

Numeral 103 denotes a CPU which relates to all the 
processings of the above-described elements. A ROM 104 
and a RAM 105 provide programs, data, work area and the 
like necessary for the processings to the CPU 103. 

15 Further, control programs necessary for the respective 
processings as shown in the flowcharts of Figs. 7 to 9 
are stored in the storage unit 107 or the ROM 104. If 
the control programs are stored in the storage unit 107, 
the programs are temporarily read into the RAM 105 then 

20 executed. 

Note that in the system configuration, various 
constituent elements other than the above-described 
elements exist, however, as the explanation of those 
elements is not a principle object of the present 

25 invention, the explanation will be omitted. 



[Image Data to Which Meta Data Is Linked] 

Figs. 2A and 2B show examples of image data to 
which meta data is linked, used in the present 
embodiment . 

5 Fig. 2A shows an example where all the meta data 

as additional information of respective image data are 
added to the end of the respective image data. Fig. 2B 
shows an example where two types of meta data exist. One 
meta data is added to the end of image data, and the 

10 other meta data is independent of the image file. The 
independent meta data has description of additional 
information of the image. On the other hand, the meta 
data added to the image data has descriptions of file 
name of the independent meta data, a file path, and a 

15 pointer indicating the location of the meta data, and 
the like. 

In the case of Fig. 2B, the additional information 
of each image data is found by a search using a meta 
data file name or a pointer indicating the location of 

20 meta data. It may be arranged such that if image data is 
selected, meta data as additional information is 
automatically selected and a keyword described in the 
meta data is displayed on the display unit 102. 

Figs. 3A to 3E are examples of meta data added to 

25 image data 1 to 5 showing only the meta data from Fig. 
2A or 2B. 



Note that in the above examples, the meta data are 
described in XML, however, the meta data may be 
described in other data description languages than XML 
such as SGML and HTML. 
5 Further, the image data 1 to 5 show examples of 

image data obtained in sports. The meta data explaining 
the contents of the respective image data include, as 
common keywords, e.g., 303 (Photographer: Takashi 
Hanamoto), 304 (Date: 2000-5-5), 305 (Location: Kamakura 
10 Dai-Ichi Junior High School), and 306 (Event: sports) in 
Fig. 3A. 

Further, as the other keywords, the meta data 
explaining the contents of the respective image data 
include, e.g., 307 (Event: opening ceremony (Fig. 3A) ) , 
15 309 (Event: opening address (Figs. 3B) ) , 311 (Event: 
relay race (Fig. 3C) ) , 313 (Event: leaving (Fig. 3D)), 
314 (Event: opening (Fig. 3E) ) , and 315 (Event: bread 
eating race (Fig. 3E) ) . 

Note that the above keywords are shown as examples, 
20 and more information may be provided. 

[Digital Album] 

A digital album is used for arrangement and 
browsing of digital images as described above. An image 
25 where a digital image is pasted on a background image 

called a template is displayed on a display screen as if 
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it is a silver-chloride photo album. 

The recording apparatus 100 of the present 
embodiment has previously-provided various scenarios (or 
themes), "sports", "wedding ceremony", "family trip", 
5 "golf competition" and the like, necessary upon 

generation of digital album as shown in Fig. 4A. Each 
scenario has plural templates. For example, the scenario 
"sports" has templates as shown in Figs. 5A to 5D. The 
scenario is stored as a template group having plural 

10 templates or a template file in the storage unit 107. 

As shown in Fig. 4B, a scenario is constructed 
with an XML file and text files, and has descriptions of 
file names of respective templates constructing the 
scenario, and locations of the files. That is, when the 

15 system reads an XML file describing a scenario, the 
system can automatically read a template group 
constructing the scenario. 

In Fig. 5A, a template 201 is an example of 
"opening ceremony"; in Fig. 5B, a template 202, an 

20 example of "bread eating race"; in Fig. 5C, a template 

203, an example of "other activities"; and in Fig. 5D, a 
template 204, an example of "closing ceremony". As shown 
in the figures, the templates respectively have 
different backgrounds and different layouts, with frames 

25 where selected images are pasted or necessary comments 
are inputted. Further, the templates are described in 
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SVG (Scalable Vector Graphics) file format as a two- 
dimensional graphics description language. Note that the 
templates may be image files such as JPEG image files. 

In the recording apparatus 100, a digital album is 
5 generated by selecting a scenario appropriate to input 
image data from the previously-provided scenarios, next, 
selecting necessary templates in accordance with the 
selected scenarios, and pasting the image data on the 
selected templates. 

10 Note that the templates as shown in Figs. 5A to 5D 

include audio data, and sound reproduction can be 
performed upon album browsing. Further, as shown in Fig. 
6, template keywords are provided for the respective 
templates, and the keywords are utilized upon comparison 

15 with meta data keywords of meta data as additional 
information of image data, for selection of optimum 
template. This operation will be described in detail in 
the flowchart of Figs. 7 to 9. 

Note that the template 203 "other activities" in 

20 Fig. 5C is a template provided for storing image data 
which does not match any of the keywords "opening 
ceremony", "bread eating race" and "closing ceremony" 
("others" template) . 

Further, as shown in Fig. 6, the template keywords 

25 have priorities. For example, the "opening ceremony" 
template 201 previously has template keywords "opening 
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ceremony", "opening", "opening address" and "leaving". 
Among these keywords, the keyword "opening ceremony" has 
the first priority, and the keyword "opening" has the 
second priority. In this manner, the templates 
5 respectively have a predetermined priority. 

Further, as shown in Figs. 5A to 5D, the templates 
have image data frames 205 to 212 where image data are 
to be pasted. Further, the size and number of image data 
frames can be increased/decreased in correspondence with 
10 image data to be pasted. 

Further, comment frames 213 to 215 are areas where 
comments on pasted image data are displayed. 

[Digital Album Generation Procedure] 
15 Fig. 7 is a flowchart showing a digital album 

generation procedure by using the recording apparatus 
100. 

First, at step S401, a scenario of digital album 
to be generated is selected from the scenarios as shown 

20 in Fig. 4A. In the present embodiment, a list of the 
scenarios in Fig. 4A is displayed on the display unit 
102, and a desired scenario is selected by using a 
pointing device of the input unit 101. When a scenario 
has been selected, plural templates linked to the 

25 selected scenario are selected. Otherwise, a template 
file linked to the selected scenario is selected. 
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Next, the process proceeds to step S4 02, at which 
a folder holding image data to be used in the album is 
selected or image data group to be used in the album is 
selected. Further, it may be arranged such that an image 
5 data group to be used in the album is previously 

designated, and when a scenario is selected, the image 
data group is automatically selected. 

Next, at step S403, image data are read one by one 
from the image data group in the selected folder or 

10 selected image data group, and the contents of meta data 
as additional information of the image data are read 
into the RAM 105. Then, a keyword matching analysis is 
performed by comparing a keyword of the read meta data 
with the respective template keywords. This processing 

15 is performed on all the image data in the selected 
folder or selected image data group. 

Next, at step S4 04, based on the result of 
matching analysis at step S403, the image data are 
classified by template. The details of the processing at 

20 steps S403 and S404 will be described in detail with 
reference to the flowchart of Fig. 8. 

Next, at step S405, the image data classified by 
template by the processing at step S404 are inserted 
into the respective image data frames. The details of 

25 the processing will be described in detail with 
reference to the flowchart of Fig. 9. 
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Next, at step S406, the generated digital album is 
saved in SVG (Scalable Vector Graphic) file format as a 
two-dimensional graphics language or an image file 
format such as JPEG, and the series of digital album 
5 generation operation is completed. In this manner, a 

digital album can be generated by a simple operation by, 
e.g., automatically pasting image data on the templates 
in Figs. 5A to 5D. Further, comments can be freely 
written in the comment frames 213 to 215 after the 
10 generation of digital album. 

[Image Data Assignment] 

Fig. 8 is a flowchart showing the image data 
classification operation at steps S403 and S404 in Fig. 
15 7. Note that as the processing shown in Fig. 8 is 

performed on meta data of one image data, the processing 
in Fig. 8 is repeated by the number of image data to be 
processed. 

First, at step S501, a keyword is read from the 
20 meta data of processed image data (e.g., any one of the 
keywords in Figs. 3A to 3E) . 

Next, at step S502, a matching analysis is 
performed between the read meta data keyword and 
keywords described in the respective templates (Fig. 6) . 
25 Next, at step S503, if it is determined as a 

result of matching analysis that the meta data keyword 
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matches some of the template keywords, the process 
proceeds to step S504, while if the meta data keyword 
does not match any template keyword, the process 
proceeds to step S505. 
5 At step S504, the number of template keywords 

matching the meta data keyword is examined. If the meta 
data keyword matches plural template keywords, the 
process proceeds to step S506, while if the meta data 
keyword matches one template keywords, the process 

10 proceeds to step S508. 

Next, at step S506, a highest priority keyword is 
searched from the obtained plural keywords, then the 
image data is assigned to the template of the highest 
priority keyword, and the series of operations is 

15 completed. 

Next, the case where the meta data explaining the 
contents of the image data 5 has two keywords "opening" 
and "bread eating race" will be described with reference 
to the example of Fig 3E. 

20 In this case, as the keywords match the template 

201 with the second priority ("opening") and the 
template 202 with the first priority ("bread eating 
race"), the templates 201 and 202 can be used. 

However, as the priority order of "bread eating 

25 race" is first in the template 202 and that of "opening" 
is the second in the template 201, the highest priority 
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keyword "bread eating race" is taken, and the keyword is 
assigned to the template 202. That is, the image data 5 
is assigned to the template 202. 

Note that if the image data frame 207 of the 
5 template 202 is already filled with a predetermined 
number of image data, the image data 5 is assigned to 
the template 202 with the next priority order keyword 
"opening" . 

The "assignment to the next highest priority 

10 template" is repeated until a template having an 

available image data frame is found. If none of the 
templates has an available image data frame, the image 
data is assigned to the "others" template. Further, if 
plural highest-priority keywords exist, the image data 

15 is assigned at random. 

Another example will be described with reference 
to Fig. 3D where the meta data explaining the contents 
of the image data 4 has a keyword "leaving". In this 
case, as the keyword matches the 4th priority keyword of 

20 the template 201 ("leaving") and the 3rd priority 
keyword of the template 204 ("leaving"), the both 
templates 201 and 204 can be used. 

However, as the priority order of the keyword 
"leaving" is the 3rd in the template 204 and the 4th in 

25 the template 201, the template 204 is used prior to the 
template 201. As a result, the keyword is assigned to 
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the template 204. That is, the image data 4 is assigned 
to the template 204. 

The "assignment to the next highest priority 
template" is repeated until a template having an 
5 available image data frame is found. If none of the 
templates has an available image data frame, the image 
data is assigned to the "others" template. Further, if 
plural keywords have the same priority order, the image 
data is assigned at random. 

10 On the other hand, if the number of templates is 

one, the process proceeds from step S506 to S508, at 
which the image data is assigned to the selected 
template, and the series of operations is terminated. 

For example, the keywords "opening ceremony" and 

15 "opening address" of the image data 1 (Fig. 3A) and the 
image data 2 (Fig. 3B) both match keywords of the 
template 201. Further, the template 201 has two image 
data frames. Accordingly, the image data 1 and the image 
data 2 are both assigned to the template 201. 

20 Next, at step S505, image data assignment is 

performed in the case where the keyword of the meta data 
does not match any of the keywords of the templates 201, 
202 and 204 in Fig. 6, as in the case of Fig. 3C . The 
image data is assigned to "other activities" of the 

25 template 203 in Fig. 6 as the "others" template, and the 
series of operations is terminated. 
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[Image Data Pasting] 

Fig. 9 is a flowchart showing a procedure of 
inserting image data, assigned to a corresponding 
5 template, into image data frames. The following 

processing is performed on all the templates included in 
the scenario. 

First, at step S601, the number of image data 
assigned to a template is examined. 
10 Next, at step S602, it is determined whether or 

not the number of assigned image data is 0. If the 
number of image data is not 0, the process proceeds to 
step S603, while if the number of image data is 0, the 
process proceeds to step S604. 
15 Next, at step S604, the template having no image 

data is removed from the album, and the series of 
operations is terminated. 

On the other hand, at step S603, the number of 
image data frames of the template is increased/decreased 
20 to be equal to the number of the assigned image data. 
Note that a predetermined upper limit is set on the 
number of image data frames so as to avoid concentration 
of image data into one template. 

Next, the process proceeds to step S605, at which 
25 it is determined whether or not the number of image data 
is greater than the predetermined upper limit value of 
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the image data frames. If the number of image data is 
not greater than the predetermined upper limit value of 
the image data frames, the process proceeds to step S606, 
at which the assigned image data are pasted on the image 
5 data frames of the respective templates as shown in Figs. 
5A to 5D. Then the series of operations is terminated. 

On the other hand, at step S605, if the number of 
image data to be pasted is greater than the upper limit 
value of the number of the image data frames, the 

10 process proceeds to step S607a, at which the image data 
are pasted, sequentially from image data having the 
highest priority keyword, on the image data frames. At 
step S607b, the remaining image data, which have not 
been pasted at step S607a, are pasted on the image data 

15 frame of the "other activities" template 203. Then the 
series of operations is terminated. 

Further, if plural image data have the same 
priority, image data are selected at random from those 
image data and pasted on the image data frames, and the 

20 remaining image data which have not been pasted are 

pasted on the image data frame of the "other activities" 
template 203. Further, if many image data are pasted on 
the "other activities" template 203 and the number of 
image data is greater than the upper limit number of the 

25 image data frames, the "other activities" template 203 

is duplicated and the remaining image data are pasted on 
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the newly-duplicated "other activities" template 203. 

In this manner, the user obtains digital album 
automatically-generated only by performing a simple 
operation, i.e., designation of desired image data group 
5 and desired template. Further, it may be arranged such 
that the keyword at the end of the meta data, e.g., the 
keyword "50th anniversary meeting" (308) in Fig. 3A or 
the keyword "last meeting before rebuilding" in Fig. 3B 
is automatically inserted in the comment frame 213. The 

10 recognition of comment is made by, e.g., in meta data 
having keywords each belonging to any of the templates, 
detecting a keyword which does not belong to any of the 
templates. Further, comments may be manually put in the 
respective comment frames 213 (Fig. 5A) with a keyboard 

15 etc. 

Further, in the above embodiment, still image data 
with meta data is handled, however, audio data with meta 
data or moving image data with meta data may be 
incorporated into a template described in SMIL 

20 (Synchronized Multimedia Integration Language) as a 

multimedia description language and used for generation 
of digital album. 

Note that SMIL is a multimedia language used in 
RealPlayer or the like for describing a method of 

25 reproduction of moving image data, audio data and still 
image data. As the method of digital album generation is 
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the same as described above using still image data, i.e., 
selecting a scenario, searching meta data for necessary- 
moving image data and/or audio data and inserting the 
data into templates, the explanation of the method will 
5 be omitted here. 

As described above, according to the present 
embodiment, a digital album can be automatically 
generated only by designating a scenario and an image 
data group using still image data with meta data. Thus, 

10 the recording apparatus and method of the present 

embodiment omit many troublesome works in generation of 
digital album. 

Further, it is preferable that a desired album can 
be easily generated only by selecting a template file 

15 constructed with plural templates. Further, in the case 
where an upper limit is set on binary data arrangeable 
in template, if the binary data are rearranged when the 
number of binary data is greater than the upper limit, 
the same template is newly generated. Accordingly, even 

20 if many meta data have a particular content and they 

cannot be arranged in one template, remaining data can 
be arranged in the newly-generated same template. 

As described above, the present invention provides 
apparatus and method for easily generating a digital 

25 album by utilizing binary data to which meta data is 
previously added. Further, the invention provides 



-22- 



apparatus and method for automatically generating a 
digital album by utilizing meta data added to binary 
data . 



5 <Second Embodiment> 

In the present embodiment, editing of moving image 
data is performed by using a personal computer, a 
specialized device or the like as a moving image data 
editing apparatus. Fig. 10 is a block diagram showing 

10 the system configuration of the moving image editing 
apparatus according to the second embodiment . 

In Fig. 10, numeral 1100 denotes a data 
input /output device which performs data input /output ; 
1101, an input device including a keyboard and a 

15 pointing device such as a mouse, which inputs 

instructions and data from the user; and 1102, a storage 
device such as a hard disk in which binary data as 
moving image data to be edited and meta data linked to 
the binary data are stored. 

20 Numeral 1103 denotes a display device which 

displays a GUI and images under the control of CPU 1104. 
As the display device 1103, a CRT, a liquid crystal 
display or the like is used. The CPU 104 relates to all 
the processings at the above-described elements. 

25 Numerals 1105 and 110 6 denote a ROM and a RAM which 

provide programs, data, work area and the like necessary 
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for various processings to the CPU 1104. Note that 
control programs necessary for the respective 
processings as shown in the flowcharts of Fig. 13 and 
the subsequent figures are stored in the storage device 
5 1102 or the ROM 1105. If the control programs are stored 
in the storage device 1102, the programs are temporarily 
read into the RAM 1106 then executed by the CPU 1104. 

Note that in the system configuration, various 
constituent elements other than the above-described 

10 elements exist, however, as the explanation of those 
elements is not a principle object of the present 
invention, the explanation will be omitted. 

Moving image editing is reading moving image data 
obtained by a digital video camera or the like into a 

15 personal computer or the like, then adding a title, 
rearranging image data, and/or processing a joint 
between images. In the moving image editing method 
according to the present embodiment, themes such as 
"sports" and "wedding ceremony" are prepared, then a 

20 search is made for image data corresponding to one of 
the themes, and the obtained moving image data are 
automatically display-described in predetermined 
templates . 

Fig. 11 is an explanatory view schematically 
25 showing moving image editing processing according to the 
present embodiment. Upon editing, first, a large frame 



-24- 



such as a theme 1201 is set. In this embodiment, as an 
example of theme, "wedding" is used. The theme 1201 
includes a theme title image 1202 and one or more 
templates 1203 and 1204. The title image 1202 displays a 
5 title corresponding to the theme 1201. The title image 
is a content displayed, e.g., in the first place, upon 
moving image reproduction. 

A template produces one partitioned scene in a 
theme. For example, in the theme "wedding", "church" of 

10 the template 1203 and "wedding party" of the template 

1204 correspond to such scenes. The template includes a 
template title image 1205, a template keyword 1209 
prepared in the template, a moving reproduction frame 
1210 and the like. The title image 1205 first displays a 

15 template title upon moving image reproduction within the 
template. 

The keyword 1209 is used for searching for moving 
image data put in the template 1204. A moving image 
search is made based on whether or not the keyword 1209 

20 matches the content of meta data registered in moving 
image data. Otherwise, a moving image search is made 
based on whether or not a feature amount linked to the 
keyword matches that of image. The moving image found by 
the search is inserted in the template. For example, 

25 assuming that a keyword "chaplain" exists in the 

template 1203, and meta data "chaplain" is registered in 
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the moving image data 1206, as the template keyword 
matches the meta data registered in the moving image 
data, the moving image data 1206 is inserted into the 
template 1203. 

5 The moving image reproduction frame 1210 is used 

as a frame for moving image reproduction or a background. 
Various patterns and characters are prepared for 
improvement in appearance upon moving image reproduction. 
Further, the templates include special effects applied 
10 to a joint of moving image data and the like. Further, 
the templates are arranged in reproduction order. For 
example, in Fig. 11, the title image 1202, then the 
template 1203, and the template 1204, are sequentially 
reproduced. 

15 Note that the reproduction order is described in 

the theme 1202. That is, in the theme 1201, descriptions 
• of the title image, the template 1203 and the template 
1204 are made in the reproduction order. Each template 
has descriptions of title image, moving images obtained 

20 by search, data reproduced in a joint between respective 
moving images {or special effect in the joint) and the 
like. 

Hereinbelow, moving image editing by utilizing a 
moving image data search using meta data will be 
25 described. Meta data means data on data, including a 

keyword for moving image data in the present embodiment. 
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Fig. 12A shows a data structure where meta data is 
added to moving image data. In Fig. 12A, an image file 
1300 has moving image data 1301 and meta data 1302 added 
to the end of the moving image data 1301. Note that in 
5 the present embodiment, moving image data with meta data 
will be referred to as a moving image file. 

The moving image data 1301 has a general data 
format having a header 1303 and data 1304. In the 
present embodiment, the meta data 1302 described in XML 
10 language includes information "location", "date", 

"object", "keyword" and the like. The meta data may have 
more information. Further, as the data description 
language in the meta data, other data description 
languages such as HTML and SGML than the XML may be used. 
15 Further, meta data may be added to plural 

positions in one moving image file. For example, in Fig. 
12B, a data portion of one image moving data is divided 
into three parts in time unit (data 1311, 1313 and 1315), 
and meta data (1312, 1314 and 1316) are added to the 
20 respective partial data. That is, the moving image file 
1310 includes a header 1320, the data portion divided 
into three partial data, and three meta data 
corresponding to the three partial data. The data 
division in moving image data is performed in time or 
25 frame units . 

In addition, it may be arranged such that meta 
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data is described in the header of moving image data, or 
meta data is held as another file. In the latter case, 
the meta data includes pointer information to specify 
image data. 

5 Hereinbelow, processing upon editing using plural 

moving image files having the above structure will be 
described. 

Fig. 13 is a flowchart showing processing upon 
moving image editing. First, at step S1401, the theme of 

10 moving image to be edited is determined. One of 

previously-provided themes such as "wedding ceremony" 
and "sports" is selected. When the theme has been 
determined, the system automatically selects a template 
group corresponding to the theme. In the present 

15 embodiment, a template group and a description of its 

reproduction order (hereinbelow referred to as a "theme 
description") are registered for each theme. When a 
theme to be used is determined, a template group to be 
used and its reproduction order are determined in 

20 accordance with the corresponding theme description. For 
example, in Fig. 11, regarding the theme "wedding 
ceremony" 1201, a template group having the templates 
1203 and 1204 and a theme description indicating that 
reproduction is made from the template 1203 to the 

25 template 1204 are determined. Note that it may be 

arranged such that editing including deletion/addition 
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of desired template and change of reproduction order can 
be performed on the theme description. 

As described in Fig. 11, each template has 
descriptions of title image, moving image reproduction 
5 frame, keyword group and the like in a multimedia 
description language such as SMIL. Note that the 
description may be made in another data description 
languages such as XML, SVG, XHTML, HTML or SGML. 

Next, at step S1402, one of the templates 
10 designated by the theme description is read from the 

storage device 1102 in Fig. 10. Then at step S1403, a 
search for moving image data is made in accordance with 
the keyword 1209 described in the template read at step 
S1402. The details of the search processing will be 
15 described later. 

At step S1404, the moving image data found by the 
search at step S14 03 are rearranged in time-sequential 
order by date. Then at step S1405, the moving image data 
rearranged at step S1404 are described in the templates 
20 so as to be displayed in the order. The details of the 
moving image data display description into the templates 
will be described later. 

At step S1406, it is determined whether or not the 
moving image data display description has been completed 
25 in all the templates in the template group described in 
the theme description. If a template where the display 
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description has not been completed exists, the process 
returns to step S1402 to repeat the above-described 
processing for the remaining templates. 

On the other hand, if the moving image data 
display description has been completed in all the 
templates, the process proceeds to step S1407, at which 
overlapped moving image data are adjusted. Overlapped 
moving image data are moving image data display- 
described in plural templates. That is, in the 
processing at step S14 07, one moving image data display- 
described in plural templates is display-described in 
one more appropriate template. 

More specifically, the date of overlapped moving 
image data is compared with the dates of previous and 
subsequent moving image data, then the display 
description of the overlapped moving image data is left 
only in a template having moving image data with closer 
date, and the other moving image data are deleted from 
the other templates. Otherwise, adjustment may be made 
by matching level between a keyword group included in 
the meta data and keyword groups described in the 
templates (the number of keywords matching the meta 
data) . 

At step S1408, the plural templates are described 
as one and outputted as a file. The file has a format as 
shown in Fig. 2, and outputted in a multimedia 
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description language such as SMIL . Another data 
description language such as XML, SVG, XHTML, HTML or 
SGML may be used. Further, the file may be outputted not 
as a data description language file but as a moving 
5 image file in format of MPEG1, MPEG2, MPEG4, AVI, 
QuickTimeMovie or the like. 

Next, the moving image data search processing at 
step S1403 will be described. Fig. 14 is a flowchart 
showing the moving image data search processing 
10 according to the present embodiment. First, at step 

51501, all the keywords described in the template are 
read and stored into the RAM 1106 in Fig. 10. At step 

51502, one of moving image files as a search object is 
read from the storage device 1102, and meta data is 

15 extracted and stored into the RAM 1106 in Fig. 10. 

At step S1503, each keyword in the template is 
compared with the meta data obtained at step S1502 to 
determine whether or not the keyword matches the meta 
data. If the keyword matches the meta data, the process 

20 proceeds to step S1504, at which a path of the moving 

image data in the storage device 1102 is stored into the 
RAM 1106, and the process proceeds to step S1505. 

Note that if the moving image file having meta 
data matching the keyword has a format as shown in Fig. 

25 12A, the path to the moving image file becomes the path 
to the moving image data. Further, if the moving image 
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file has a structure as shown in Fig. 12B, and the meta 
data matches the keyword of partial data divided from 
one moving image data, the position of the partial data 
in the moving image file is stored into the RAM 1106. 
5 The designation of the position of the partial data in 
the moving image file may be e.g. designation of time 
(minutes and seconds) from the start position of the 
moving image data. Otherwise, the number of frames from 
the start position may be designated. Note that at step 

10 S1503, if the keyword does not matches the meta data, 
the process proceeds to step S1505. 

At step S1505, it is determined whether or not the 
processing at steps S1502 to S1504 has been performed on 
all the moving image data in the storage device 1102. If 

15 there is unprocessed moving image data, the process 

returns to step S1502, at which the above processing is 
repeated for the remaining moving image data. On the 
other hand, if the above processing has been performed 
on all the moving image data, the process proceeds to 

20 step S1506. At step S1506, it is determined whether or 
not a path to the moving image data exists in the RAM 
1106. If no path exists in the RAM, the process proceeds 
to step S1507, at which the template is deleted from the 
theme description since the template has no moving image 

25 data to be reproduced, and the search operation is 

terminated. On the other hand, if a path exists, the 
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search operation is terminated after step S1506. 

Note that in the above processing, an image file 
including moving image data and meta data as shown in 
Figs. 12A and 12B is handled as a search object, however, 
5 if moving image data and meta data are separately held, 
a search is made for meta data matching the keyword at 
step S1502, then at step 1504, a path to moving image 
data registered in the meta data detected by the search 
is stored into the RAM 1106. 
10 Next, the processing of display-describing moving 

image data into the templates at step S1405 in Fig. 13 
will be described with reference to the flowchart of Fig. 
15. 

At step S1601, the moving image data arranged in 
15 the time-sequential order by date are display-described 
in the templates such that the image data are to be 
reproduced in the order. The dates are obtained from 
information recorded in the meta data or the headers. At 
step S1602, in description of plural moving image data, 
20 special effect processing is performed on a joint 

between the moving image data for smooth scene change. 
The special effect processing is description in 
accordance with the theme in a description language such 
as SMIL into the template. For example, if the theme is 
25 "wedding ceremony", an animation of crackers of 

congratulation is prepared as a special effect, and the 
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moving image data changes before or after the special 
effect. Further, general effects such as dissolve and 
cross may be used. These special effect processing are 
previously provided in a description language such as 
5 SMIL in the template, and automatically added to the 
description. 

At step SI 603, information of time, location and 
the like are described in the templates so as to be 
displayed on the display screen. Then the process ends. 

10 By this processing, when a predetermined operation (e.g., 
clicking a predetermined position by a mouse) is 
performed, meta data on time, location and the like can 
be displayed on the display screen. It may be arranged 
such that these data are always displayed. 

15 As described above, according to the present 

embodiment, moving image data are selected in accordance 
with a keyword for moving image search described in a 
template, and display description is made in the 
template to display the selected moving image data, 

20 thereby editing of moving image data is realized. The 

user merely selects a theme including templates, and the 
editing operation is extremely easily realized. 

Further, if plural moving image data are to be 
display-described in one template, as predetermined 

25 processing is performed on a joint between different 

moving image data, an editing operation on the joint can 
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be omitted. 

Further, if plural moving image data are to be 
displayed in one template, as the image data are 
arranged such that reproduction is made in the time- 
5 sequential order by date of generation of data, image 
reproduction can be performed in natural order. 

Further, one theme is selected, and plural 
templates related to the theme are automatically 
selected. Then editing of moving image data is performed 
10 in the respective selected plural templates. Accordingly, 
the user only selects a theme and the moving image 
editing is automatically performed, thereby troublesome 
operations are omitted. 

Further, at this time, if image data are display- 
15 described in plural templates, only one template in 
which the moving image data is to be described is 
automatically selected so as to avoid display 
description of the moving image data in other templates . 
Thus overlapped reproduction of image data is prevented, 
20 and more appropriate editing is realized. 

Further, as shown in Fig. 12B, meta data is added 
to each partial data divided from moving image data, and 
image data editing is performed in partial data units. 
Thus more elaborate image editing is realized. 

25 

<Third Embodiment> 
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In the second embodiment, the moving image data 
search is performed by using meta data. In the present 
embodiment, the moving image data search is performed by- 
similar image search and moving image editing is 
5 performed. 

Similar image search is a technique of dividing an 
image into arbitrary number of images, extracting 
feature amount of each image (color and its location) 
and performing an image search. Note that as the present 
10 embodiment has the same construction and processing 

procedures as those in the second embodiment except the 
moving image data search, the explanations of the 
construction and procedures will be omitted. 

Fig. 16 is a flowchart showing the moving image 
15 data search processing by utilizing the similar image 
search. Note that in Fig. 16, steps S1701, S1705 to 
S1707 are identical to steps S1501, S1505 to S1507 in 
Fig. 14. 

At step S1701, all the keywords are read from a 
20 template and stored into the RAM 1106 in Fig. 10. At 

step S1702, one of the plural moving image data is read 
from the storage device 1102 in Fig. 10, and images are 
extracted at predetermined frame intervals from the 
moving image data. Then, feature amounts of all the 
25 extracted images are calculated and stored into the RAM 
1106 in Fig. 10. Note that if the moving image data is 
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stored as a moving image file as shown in Fig. 12A or 
12B, the moving image data is extracted from the moving 
image file, and frame extraction and feature amount 
calculation are performed. 
5 Next, at step S1703, the keyword of the template 

is compared with the feature amounts of the moving image 
data, to find a match. Note that the keyword is 
converted into an image feature amount by referring to a 
table of image feature amounts corresponding to 

10 respective keywords previously stored in the storage 

device 1102. For example, the system converts a template 
keyword "sunset" by referring to the table into a 
feature amount of "an image having an orange to yellow 
object at the center in a reddish background". Then the 

15 obtained feature amount is compared with each of the 

feature amounts stored into the RAM 1106 at step S1702. 

If the feature amounts match or comparatively 
close to each other (fuzzy search) , the process proceeds 
to step S1704. If the feature amounts do not match or 

20 not comparatively close to each other, the process 

proceeds to step S1705. At step S1704, the path of the 
moving image data in the storage device 1102 is stored 
into the RAM 1106. At step S1705, it is determined 
whether or not the processing at steps S1702 to S1704 

25 has been performed on all the moving image data in the 

storage device 1102. If there is unprocessed image data, 



-37- 



the process returns to step S1702, at which the above 
processing is performed on the remaining moving image 
data. 

If the above processing has been performed on all 
the moving image data, the process proceeds to step 

51706. At step S1706, it is determined whether or not a 
path to the moving image data exists in the RAM 1106. If 
no path exists, the process proceeds to step S1707. If a 
path exists, the search operation is terminated. At step 

51707, the template having no moving image data matching 
the feature amount is deleted from the theme description, 
and the search operation is terminated. 

As described above, according to the above second 
and third embodiments, moving image editing can be 
automatically performed by utilizing templates and 
moving image search. The user only selects a theme, and 
moving image editing is automatically performed. Thus 
troublesome works can be omitted. 

Note that in the third embodiment, the scenic word 
"sunset" is used as the template keyword, however, other 
types of words e.g. a word directly expressing the color 
of image such as "red" or "orange" may be used. Further, 
it may be arranged such that a template has an image of 
"sunset" and a search is made for moving image data 
similar to the image. 

Further, in the above embodiment, the feature 
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amount of image is determined by the color and location 
in the image, however, it may be arranged such that the 
an outline is extracted from the image and the feature 
amount is determined based on the outline. 
5 Further, in addition to the moving image data 

search described in the present embodiment, following 
moving image data search may be performed. For example, 
items "amount of movement (change) of moving image" or 
"level of sound" is provided in templates, and the value 

10 is set in the respective templates. The amount of 

movement or the level of sound of the moving image data 
in the moving image file is actually examined, and a 
search is made for data having a value close to the set 
value. Otherwise, it may be arranged such that audio 

15 data added to moving image data is recognized, and the 
audio data is compared with a template keyword. 

Note that the classification into templates by 
similar image search in the third embodiment is 
applicable to the still image in Fig. 1. 

20 As described above, according to the present 

invention, moving image editing can be automated, and 
moving image data can be easily edited. 

Note that the object of the present invention can 
be also achieved by providing a storage medium {or 

25 recording medium) storing software program code for 
performing the aforesaid processes to a system or an 
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apparatus, reading the program code with a computer 
(e.g., CPU, MPU) of the system or apparatus from the 
storage medium, then executing the program. 

In this case, the program code read from the 
storage medium realizes the functions according to the 
embodiments, and the storage medium storing the program 
code constitutes the invention. 

Further, the storage medium, such as a floppy disk, 
a hard disk, an optical disk, a magneto-optical disk, a 
CD-ROM, a CD-R, a DVD, a magnetic tape, a non-volatile 
type memory card, and ROM can be used for providing the 
program code . 

Furthermore, besides aforesaid functions according 
to the above embodiments are realized by executing the 
program code which is read by a computer, the present 
invention includes a case where an OS (operating system) 
or the like working on the computer performs a part or 
entire processes in accordance with designations of the 
program code and realizes functions according to the 
above embodiments . 

Furthermore, the present invention also includes a 
case where, after the program code read from the storage 
medium is written in a function expansion card which is 
inserted into the computer or in a memory provided in a 
function expansion unit which is connected to the 
computer, CPU or the like contained in the function 
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expansion card or unit performs a part or entire process 
in accordance with designations of the program code and 
realizes functions of the above embodiments. 

As many apparently widely different embodiments of 
5 the present invention can be made without departing from 
the spirit and scope thereof, it is to be understood 
that the invention is not limited to the specific 
embodiments thereof except as defined in the appended 
claims . 
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